Methods and systems for generating an animation control rig including posing of non-rigid areas

ABSTRACT

In an embodiment, an animator is provided with an indication when a model&#39;s component such as a joint or limb is being moved or twisted in a way that would be unnatural and cause unusual stress on the model component. For example, as a shoulder joint is stressed by moving an arm in an extreme position a yellow bar or coloring of the shoulder, arm or other component can grow increasingly bright and shift to red just before a breaking point is reached. An animator can choose to go past the breaking point and the breaking can be modeled and incorporated into the animation.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of the following application, U.S.patent application Ser. No. 17/120,065, entitled METHODS AND SYSTEMS FORGENERATING AN ANIMATION CONTROL RIG, filed on Dec. 11, 2020, whichclaims the benefit of U.S. Provisional Patent Application Ser. No.63/056,420, entitled METHODS AND SYSTEMS FOR GENERATING AN ANIMATIONCONTROL RIG, filed on Jul. 24, 2020, which is hereby incorporated byreference as if set forth in full in this application for all purposes.

FIELD

The present disclosure generally relates to methods and systems forgenerating an animation using an indication of a stress level beingexerted on a component of an animation model.

BACKGROUND

Visual content generation systems are used to generate imagery in theform of still images and/or video sequences of images. The still imagesand/or video sequences of images include live action scenes obtainedfrom a live action capture system, computer generated scenes obtainedfrom an animation creation system, or a combination thereof.

An animation artist is provided with tools that allow them to specifywhat is to go into that imagery. Where the imagery includes computergenerated scenes, the animation artist may use various tools to specifythe positions in a scene space such as a three-dimensional coordinatesystem of objects. Some objects are articulated, having multiple limbsand joints that are movable with respect to each other.

The animation artist may retrieve a representation of an articulatedobject and generate an animation sequence movement of the articulatedobject, or part thereof. Animation sequence data representing ananimation sequence may be stored in data storage, such as animationsequence storage described below.

Animation sequence data might be in the form of time series of data forcontrol points of an articulated object having attributes that arecontrollable. Generating animation sequence data has the potential to bea complicated task when a scene calls for animation of an articulatedobject.

SUMMARY

In accordance with an aspect, a computer-implemented method for creatingan animation includes: displaying an animation model including acomponent; displaying a control point corresponding to at least onecomponent of the animation model; accepting a signal from a user inputdevice to move the control point; determining a stress amount inresponse to the moved control point; indicating to a user the stressamount; and creating an animation using a movement derived from themoved control point.

The term ‘comprising’ as used in this specification means ‘consisting atleast in part of’. When interpreting each statement in thisspecification that includes the term ‘comprising’, features other thanthat or those prefaced by the term may also be present. Related termssuch as ‘comprise’ and ‘comprises’ are to be interpreted in the samemanner.

In an embodiment, the method further includes: determining a stresslevel of a component of the digital character resulting from movementtoward the new pose; and displaying an indicator to visually indicatethe stress level.

In an embodiment, the visual indicator includes a color.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 shows an example of a control rig configured to enable an artistto create animation sequence data.

FIG. 2 shows an example of an animation skeleton obtained from ananimation system that is matched to the skeleton of the control rig ofFIG. 1.

FIG. 3 shows examples of animation control points associated with thecontrol rig of FIG. 1.

FIG. 4 shows an example of a hierarchical node graph suitable forimplementing the control rig of FIG. 1.

FIG. 5 shows an example method for generating the animation control rigof FIG. 1.

FIG. 6 is a block diagram illustrating an example computer system uponwhich computer systems of the systems illustrated herein may beimplemented.

FIG. 7 illustrates an example visual content generation system as mightbe used to generate imagery in the form of still images and/or videosequences of images.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

Described below are methods and systems for generating an animationcontrol rig configured to manipulate a skeleton of a digital character(an “animation skeleton”).

FIG. 1 shows an example of a control rig 100, or animated skeleton.Control rig 100 is configured to enable an artist to create animationsequence data. Animation sequence data is typically in the form of timeseries of data for control points of an object that has attributes thatare controllable. In some examples the object includes a humanoidcharacter with limbs and joints that are movable in manners similar totypical human movements.

Here, control rig 100 represents a humanoid character, but may beconfigured to represent a plurality of different characters. In anembodiment control rig 100 includes a hierarchical set of interconnectedbones, connected by joints forming a kinematic chain.

For example, control rig 100 includes thigh 102, knee 104, lower leg106, ankle 108, and foot 110, connected by joints 112, 114 to form a legappendage 118. Control rig 100 may be employed to individually moveindividual bones and joints using forward kinematics to pose acharacter. Moving thigh 102 causes a movement of lower leg 106, as lowerleg 106 is connected to thigh 102 via knee 104. Thigh 102 and lower leg106, for example, are in a parent-child relationship. Movement of lowerleg 106 is a product of movement of thigh 102 as well as movement oflower leg 106 itself. Control rig 100 may also use inverse kinematics,in which an artist moves ankle 108 for example. If an artist moves ankle108 upwards, knee 104 consequently bends and moves upwards toaccommodate a pose in which ankle 108 is at a user specified location.

Control rig 100 may be formed using a plurality of data points. Controlrig 100 may be matched to a skeleton obtained from an animation system,or from, for example, motion capture markers or other means on real-lifeactors. A live action scene of a human actor is captured by live actioncapture system 702 (see FIG. 7) while wearing mo-cap fiducials forexample high-contrast markers outside actor clothing. The movement ofthose fiducials is determined by live action processing system 722.Animation driver generator 744 may convert that movement data intospecifications of how joints of an articulated character are to moveover time.

FIG. 2 shows an example of a skeleton 200 obtained from an animationsystem such as visual content generation system 700.

The motions of control rig 100 are able to correspond to the motions ofmotion captured skeleton 200 when matched. Control rig 100 may also becontrolled freely by an animator to produce motions beyond the motionsof a real-life skeleton, such as the real-life skeleton of a human.Control rig 100 may represent a character of a different size to theskeleton of a real-life actor.

As shown in FIG. 3, control rig 100 includes a plurality of animationcontrol points, or control points. Examples of control points areindicated at 120, 122 and 124 respectively. For example, in anembodiment control rig 100 includes control point 120 at the ankle thatallows an animator to control the motion of a leg of control rig 100. Inanother example, control point 122 is positioned at lower leg 106 of rig100 and/or control point 124 is positioned at thigh 102. Different partsof the control rig 100 have associated to them respective controlpoints.

In an embodiment, an artist may create an animation sequence byselecting a control point on control rig 100. Control rig 100 may bedisplayed, for example, on display 612 (see FIG. 6). The artist selectsa control point using input device 614 and/or cursor 616. The controlpoints may be displayed as extending from a character represented bycontrol rig 100. Displaying the control points in this manner enablesthe artist to select a control point easily.

In an embodiment, operating animation control(s) causes an animatedobject to be generated. The animated object may be displayed, forexample, on display 612 as an image sequence.

The artist may, for example, select control point 122 for the lower legor control point 124 for the upper leg of control rig 100. The artistselects a position and/or location of the control point that isdifferent to the current position and/or location of the control point.This process is known as key-framing The artist moves controls to newpositions at given times, thereby creating key poses in an animationsequence. Interpolation is performed between key poses.

In different embodiments, different types of controls can be madeavailable to the artist. Typically, graphical user interface (GUI)controls called “handles” are presented to an artist for manipulation. Ahandle might correspond to movement of a part of a finger, for example.Or a foot, arm bone or any other component or set of components in ananimation skeleton or other underlying rig. Usually, the handles aresimplified subsets of the component definitions and control points andwill not all line up exactly with skeleton components. In otherembodiments, there can be a one-to-one correspondence of one or morehandles to control points. Or the control points themselves may bepresented to an artist or animator for manipulation.

In an embodiment, there can be one-off handles or other controls (e.g.,“control shapes”) that can allow custom movement of animation models forcomplex or non-rigid structures such as for a fat belly, a tail, lips,etc. Often animation of caricatures will not closely follow a skeletonand so handles are adapted to handle areas of motion, or other organizedsystems of movement. Animating face models may not require an underlyingskeleton in order to animate the model, or digital character.

In an embodiment, an animator is provided with a visual indication whena joint or limb is being moved or twisted in a way that would beunnatural and cause unusual stress on the model component, were thedigital character being modeled to exist in the real world. So, forexample, as a shoulder joint is stressed by moving an arm in an extremeposition a yellow bar or coloring of a model component can growincreasingly bright and shift to red just before a breaking point orfailure point is reached. An animator can choose to go past the breakingpoint and the breaking can be modeled and incorporated into theanimation.

In an embodiment, control points may be used to control more than onebone, joint, etc. For example, a control point may be used to controlthe upper arm and lower arm at the same time.

In an embodiment, at least one inverse kinematics operation is performedin order to generate the animation sequence specified by the artist. Forexample, the artist may wish to specify that ankle 108 is to move from alocation within control rig 100 shown in FIG. 1 to a location withincontrol rig shown in FIG. 3. The artist manipulates control point 120 tospecify a desired change in ankle location.

A series of calculations is performed to determine what changes inlocation and/or orientation of parts of control rig 100 are required toresult in an orientation of control rig shown in FIG. 3. For example,the new location of control point 120 selected by the artist may requirea change in location and/or orientation of at least thigh 102, knee 104,lower leg 106, ankle 108 and foot 110. The changes in location and/ororientation that are required to achieve a goal of the artist are thendetermined.

FIG. 4 shows an example of a hierarchical node graph 400 suitable forimplementing control rig 100 of FIG. 1. Node graph 400 includes aplurality of nodes, examples of which are shown at 402, 404 and 406respectively. At least some of the nodes are associated with at leastone input and at least one output.

In an embodiment one or more nodes of the hierarchical node graph 400represent respective animation control points of control rig 100.Outputs from individual nodes include the solved positions of each jointangle and bone position in the kinematic chain. In inverse kinematics,the new joint angles and positions are determined relative to thecontrol input. Inputs to the individual nodes include the new positionof a member that is then used to calculate the position of the othermembers of the skeleton and the associated joint angles. For example,moving the hand from a first position resting on the ground to a newposition above the ground will be used to determine the position of theforearm, upper arm, and elbow, and shoulder.

In an embodiment, at least one node in hierarchical node graph 400 isinversely solvable through an analytical approach. For example, node 404has associated to it an inverse kinematics function. The artist selectsa position and/or location of control point 120 that is different to thecurrent position and/or location of the control point. Node 402 inhierarchical node graph 400 is identified as corresponding to controlpoint 120. The inverse kinematics function associated to node 402 isapplied to control point 120. The output of node 402 becomes an input tonode 404. The output of node 404 then becomes the input of node 406.

The result is that the position and angle associated with each nodefurther away from the control point allows the position and/or locationof control point 120 to correspond to the position and/or locationselected by the artist. In an example, a node that is inversely solvablemay involve a problem that has three points of a limb that areanalytically solvable using a single solution, such as a trigonometricsolution.

In an embodiment at least one node in hierarchical node graph 400 is notinversely solvable through an analytical approach. For those nodes thatare not inversely solvable, there is no associated inverse kinematicsfunction. An inverse kinematics function cannot be applied to a controlpoint so that the position and/or location of the control pointcorresponds to a position and/or location selected by the artist.

In an embodiment control rig 100 and hierarchical node graph 400 aresubject to at least one constraint. For example, there may be a finitenumber of positions and/or orientations that an artist may select forknee 104. Some positions may violate physical constraints that apply toa physical skeleton associated to control rig 100. These physicalconstraints include biomechanical limitations of movement that apply toa physical skeleton and that are modeled by control rig 100.

Furthermore, some positions may involve applying transformations to atleast one node in the hierarchical node graph 400 for which there is noassociated inverse kinematics function. Such transformations violate acomputational constraint. For example, there may be cases where a nodedoes not have an inverse kinematics function.

In an embodiment, a series of forward kinematics operations are appliedto at least one node in hierarchical node graph 400 for which there isno associated inverse kinematics function. The resulting location iscompared to the desired location selected by the artist. Any differencein location is compared to a tolerance. If the difference is within thetolerance then the transformation is completed. On the other hand, ifthe difference in locations is outside the tolerance then a furtherforward kinematics operation is performed on the node with the intentionof achieving a difference between the calculated location and theselected location that is within the tolerance.

In an embodiment, an iterative solve is required where there areoverlapping solves associated with a change in a control node. It ispossible that some nodes may have more than one associated inversekinematics function related to multiple parent nodes. Most of the solvesare based on trigonometry, so there may be more than one way to solvethe equation.

Trigonometric solutions are generally directly solvable. A problem mayarise where there are solves that require more than one solve at a time.For example, a solve may be straightforward to determine for a handposition, elbow position, and shoulder position if the hand was fixedrelative to the forearm. In other words, a problem that involves threepoints of a limb is analytically solvable using a single trigonometricsolution.

However, the artist may wish to, for example, bend a wrist back and thenmove the lower arm. There is a choice of trigonometric solutions. Afirst solution may involve the bent position of the hand, forearm,elbow, and upper arm. A second solution may involve the bent position ofthe hand and just the end of the forearm. In other words, when more thanthree points of a limb are involved in a problem, multiple trigonometricsolutions involving the same points of a limb may exist. Approximationinvolving iterative solves may be required in such cases.

A control node selected by the artist may influence multiple parentnodes. The relationship between the control node and each of themultiple parent nodes has an associated inverse kinematics function.Therefore, a changed position and/or orientation of the control nodewould require multiple inverse kinematics functions to be solvediteratively to determine the positions and/or orientations of themultiple parent nodes.

In an example, when a series of movements of control rig 100 is requiredto match with a series of movements from a motion captured skeleton, thecontrol points of control rig 100 are controlled by the correspondingpositions of the motion captured skeleton. For example, an ankle of themotion captured skeleton drives a corresponding ankle of the controlrig. The inverse kinematics operations associated with the ankle node ofthe control rig are then used to derive the positions and/ororientations of at least thigh 102, knee 104 and lower leg 106.

FIG. 5 shows an example method 500 for generating animation control rig100 of FIG. 1. Method 500 uses inverse solves at an appendage levelwhere possible to determine where controls should be placed andoriented. In an embodiment, control rig 100 is configured to manipulatea skeleton of an animated character.

A plurality of animation control points is associated 502 to an animatedskeleton. One example of an animated skeleton is motion capturedskeleton 200 (see FIG. 2). Examples of control points include controlpoints 120, 122, and 124 (see FIG. 3). In an embodiment at least one ofthe control points is associated to at least two members of skeleton200. For example, a single control point may be associated to at leasttwo bones of skeleton 200.

Method 500 includes traversing a node graph representing the pluralityof animation control points of the animated skeleton. One example of anode graph is hierarchical node graph 400 (see FIG. 4) in which nodes innode graph 400 are associated with respective control points such as120, 122 and 124. It will be appreciated that method 500 can be appliedto either directed or undirected graphs, and can be applied to weightedor unweighted graphs.

A first node in node graph 400 is selected 504 for analysis to determineif the first node is inversely solvable. In an embodiment, selecting anode in node graph 400 has the effect of selecting an animation controlpoint represented by the selected node.

If 506 the selected node is inversely solvable then an inverse solve isperformed 508 to determine where the animation control point representedby the selected node should be placed and oriented. An operation of theselected node is modified by updating 510 the active skeleton.

In an embodiment, if the selected node is not inversely solvable then anapproximation 509 of an inverse solve is obtained. Method 500 attemptsto perform a solve as close as possible to the desired position. Forexample, a desired limb position may not be achievable within thelimitations of an animated skeleton. In such cases, a limb position canbe determined, within the limitations of the animated skeleton, that isas close as possible to the desired limb position. This approximationmay be obtained, for example, using an iterative solve. An operation ofthe selected node is modified by updating 510 the active skeleton withthe approximation.

Once the animated skeleton has been updated 510, a stopping condition ischecked 512.

In an embodiment, method 500 is performed at an appendage level.Stopping condition 512 is satisfied when all appendages have beenchecked. Alternatively, a stopping condition is satisfied when apredetermined number of appendages have been checked. Alternatively, astopping condition is satisfied when a sufficient number of appendageswithin a predetermined region of the animated skeleton have beenchecked.

According to one embodiment, the techniques described herein areimplemented by one or generalized computing systems programmed toperform the techniques pursuant to program instructions in firmware,memory, other storage, or a combination. Special-purpose computingdevices may be used, such as desktop computer systems, portable computersystems, handheld devices, networking devices or any other device thatincorporates hard-wired and/or program logic to implement thetechniques.

For example, FIG. 6 is a block diagram that illustrates computer system600 upon which visual content generation system 700 (see FIG. 7) may beimplemented. Computer system 600 includes bus 602 or other communicationmechanism for communicating information, and processor 604 coupled withbus 602 for processing information. Processor 604 may be, for example, ageneral purpose microprocessor.

The computer system 600 also includes main memory 606, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 602for storing information and instructions to be executed by the processor604. The main memory 606 may also be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by the processor 604. Such instructions,when stored in non-transitory storage media accessible to the processor604, render the computer system 600 into a special-purpose machine thatis customized to perform the operations specified in the instructions.

The computer system 600 further includes a read only memory (ROM) 608 orother static storage device coupled to the bus 602 for storing staticinformation and instructions for the processor 604. A storage device610, such as a magnetic disk or optical disk, is provided and coupled tothe bus 602 for storing information and instructions.

The computer system 600 may be coupled via the bus 602 to a display 612,such as a computer monitor, for displaying information to a computeruser. An input device 614, including alphanumeric and other keys, iscoupled to the bus 602 for communicating information and commandselections to the processor 604. Another type of user input device is acursor control 616, such as a mouse, a trackball, or cursor directionkeys for communicating direction information and command selections tothe processor 604 and for controlling cursor movement on the display612. This input device typically has two degrees of freedom in two axes,a first axis (e.g., x) and a second axis (e.g., y), that allows thedevice to specify positions in a plane.

The computer system 600 may implement the techniques described hereinusing customized hard-wired logic, one or more ASICs or FPGAs, firmwareand/or program logic which in combination with the computer systemcauses or programs the computer system 600 to be a special-purposemachine. According to one embodiment, the techniques herein areperformed by the computer system 600 in response to the processor 604executing one or more sequences of one or more instructions contained inthe main memory 606. Such instructions may be read into the main memory606 from another storage medium, such as the storage device 610.Execution of the sequences of instructions contained in the main memory606 causes the processor 604 to perform the process steps describedherein. In alternative embodiments, hard-wired circuitry may be used inplace of or in combination with software instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may includenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as the storage device 610.Volatile media includes dynamic memory, such as the main memory 606.Common forms of storage media include, for example, a floppy disk, aflexible disk, hard disk, solid state drive, magnetic tape, or any othermagnetic data storage medium, a CD-ROM, any other optical data storagemedium, any physical medium with patterns of holes, a RAM, a PROM, anEPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire, and fiber optics, including thewires that include the bus 602. Transmission media can also take theform of acoustic or light waves, such as those generated duringradio-wave and infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to the processor 604 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over anetwork connection. A modem or network interface local to the computersystem 600 can receive the data. The bus 602 carries the data to themain memory 606, from which the processor 604 retrieves and executes theinstructions. The instructions received by the main memory 606 mayoptionally be stored on the storage device 610 either before or afterexecution by the processor 604.

The computer system 600 also includes a communication interface 818coupled to the bus 602. The communication interface 818 provides atwo-way data communication coupling to a network link 820 that isconnected to a local network 822. For example, the communicationinterface 818 may be an integrated services digital network (ISDN) card,cable modem, satellite modem, or a modem to provide a data communicationconnection to a corresponding type of telephone line. Wireless links mayalso be implemented. In any such implementation, the communicationinterface 818 sends and receives electrical, electromagnetic, or opticalsignals that carry digital data streams representing various types ofinformation.

The network link 820 typically provides data communication through oneor more networks to other data devices. For example, the network link820 may provide a connection through the local network 822 to a hostcomputer 824 or to data equipment operated by an Internet ServiceProvider (ISP) 826. The ISP 826 in turn provides data communicationservices through the world wide packet data communication network nowcommonly referred to as the “Internet” 828. The local network 822 andInternet 828 both use electrical, electromagnetic, or optical signalsthat carry digital data streams. The signals through the variousnetworks and the signals on the network link 820 and through thecommunication interface 818, which carry the digital data to and fromthe computer system 600, are example forms of transmission media.

The computer system 600 can send messages and receive data, includingprogram code, through the network(s), the network link 820, andcommunication interface 818. In the Internet example, a server 830 mighttransmit a requested code for an application program through theInternet 828, ISP 826, local network 822, and communication interface818. The received code may be executed by the processor 604 as it isreceived, and/or stored in the storage device 610, or other non-volatilestorage for later execution.

For example, FIG. 7 illustrates the example visual content generationsystem 700 as might be used to generate imagery in the form of stillimages and/or video sequences of images. The visual content generationsystem 700 might generate imagery of live action scenes, computergenerated scenes, or a combination thereof. In a practical system, usersare provided with tools that allow them to specify, at high levels andlow levels where necessary, what is to go into that imagery. Forexample, a user might be an animation artist (like the artist 142illustrated in FIG. 1) and might use the visual content generationsystem 700 to capture interaction between two human actors performinglive on a sound stage and replace one of the human actors with acomputer-generated anthropomorphic non-human being that behaves in waysthat mimic the replaced human actor's movements and mannerisms, and thenadd in a third computer-generated character and background sceneelements that are computer-generated, all in order to tell a desiredstory or generate desired imagery.

Still images that are output by the visual content generation system 700might be represented in computer memory as pixel arrays, such as atwo-dimensional array of pixel color values, each associated with apixel having a position in a two-dimensional image array. Pixel colorvalues might be represented by three or more (or fewer) color values perpixel, such as a red value, a green value, and a blue value (e.g., inRGB format). Dimension of such a two-dimensional array of pixel colorvalues might correspond to a preferred and/or standard display scheme,such as 1920 pixel columns by 1280 pixel rows Images might or might notbe stored in a compressed format, but either way, a desired image may berepresented as a two-dimensional array of pixel color values. In anothervariation, images are represented by a pair of stereo images forthree-dimensional presentations and in other variations, some or all ofan image output might represent three-dimensional imagery instead ofjust two-dimensional views.

A stored video sequence might include a plurality of images such as thestill images described above, but where each image of the plurality ofimages has a place in a timing sequence and the stored video sequence isarranged so that when each image is displayed in order, at a timeindicated by the timing sequence, the display presents what appears tobe moving and/or changing imagery. In one representation, each image ofthe plurality of images is a video frame having a specified frame numberthat corresponds to an amount of time that would elapse from when avideo sequence begins playing until that specified frame is displayed. Aframe rate might be used to describe how many frames of the stored videosequence are displayed per unit time. Example video sequences mightinclude 24 frames per second (24 FPS), 50 FPS, 140 FPS, or other framerates. In some embodiments, frames are interlaced or otherwise presentedfor display, but for the purpose of clarity of description, in someexamples, it is assumed that a video frame has one specified displaytime and it should be understood that other variations are possible.

One method of creating a video sequence is to simply use a video camerato record a live action scene, i.e., events that physically occur andcan be recorded by a video camera. The events being recorded can beevents to be interpreted as viewed (such as seeing two human actors talkto each other) and/or can include events to be interpreted differentlydue to clever camera operations (such as moving actors about a stage tomake one appear larger than the other despite the actors actually beingof similar build, or using miniature objects with other miniatureobjects so as to be interpreted as a scene containing life-sizedobjects).

Creating video sequences for story-telling or other purposes often callsfor scenes that cannot be created with live actors, such as a talkingtree, an anthropomorphic object, space battles, and the like. Such videosequences might be generated computationally rather than capturing lightfrom live scenes. In some instances, an entirety of a video sequencemight be generated computationally, as in the case of acomputer-animated feature film. In some video sequences, it is desirableto have some computer-generated imagery and some live action, perhapswith some careful merging of the two.

While computer-generated imagery might be creatable by manuallyspecifying each color value for each pixel in each frame, this is likelytoo tedious to be practical. As a result, a creator uses various toolsto specify the imagery at a higher level. As an example, an artist(e.g., the artist 142 illustrated in FIG. 1) might specify the positionsin a scene space, such as a three-dimensional coordinate system, ofobjects and/or lighting, as well as a camera viewpoint, and a cameraview plane. Taking all of that as inputs, a rendering engine may computeeach of the pixel values in each of the frames. In another example, anartist specifies position and movement of an articulated object havingsome specified texture rather than specifying the color of each pixelrepresenting that articulated object in each frame.

In a specific example, a rendering engine performs ray tracing wherein apixel color value is determined by computing which objects lie along aray traced in the scene space from the camera viewpoint through a pointor portion of the camera view plane that corresponds to that pixel. Forexample, a camera view plane might be represented as a rectangle havinga position in the scene space that is divided into a grid correspondingto the pixels of the ultimate image to be generated, and if a raydefined by the camera viewpoint in the scene space and a given pixel inthat grid first intersects a solid, opaque, blue object, that givenpixel is assigned the color blue. Of course, for moderncomputer-generated imagery, determining pixel colors—and therebygenerating imagery—can be more complicated, as there are lightingissues, reflections, interpolations, and other considerations.

As illustrated in FIG. 7, a live action capture system 702 captures alive scene that plays out on a stage 704. The live action capture system702 is described herein in greater detail, but might include computerprocessing capabilities, image processing capabilities, one or moreprocessors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown.

In a specific live action capture system, cameras 706(1) and 706(2)capture the scene, while in some systems, there might be other sensor(s)708 that capture information from the live scene (e.g., infraredcameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).On the stage 704, there might be human actors, animal actors, inanimateobjects, background objects, and possibly an object such as a greenscreen 710 that is designed to be captured in a live scene recording insuch a way that it is easily overlaid with computer-generated imagery.The stage 704 might also contain objects that serve as fiducials, suchas fiducials 712(1)-(3), that might be used post-capture to determinewhere an object was during capture. A live action scene might beilluminated by one or more lights, such as an overhead light 714.

During or following the capture of a live action scene, the live actioncapture system 702 might output live action footage to a live actionfootage storage 720. A live action processing system 722 might processlive action footage to generate data about that live action footage andstore that data into a live action metadata storage 724. The live actionprocessing system 722 might include computer processing capabilities,image processing capabilities, one or more processors, program codestorage for storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. The live action processing system 722 mightprocess live action footage to determine boundaries of objects in aframe or multiple frames, determine locations of objects in a liveaction scene, where a camera was relative to some action, distancesbetween moving objects and fiducials, etc. Where elements are sensoredor detected, the metadata might include location, color, and intensityof the overhead light 714, as that might be useful in post-processing tomatch computer-generated lighting on objects that are computer-generatedand overlaid on the live action footage. The live action processingsystem 722 might operate autonomously, perhaps based on predeterminedprogram instructions, to generate and output the live action metadataupon receiving and inputting the live action footage. The live actionfootage can be camera-captured data as well as data from other sensors.

An animation creation system 730 is another part of the visual contentgeneration system 700. The animation creation system 730 might includecomputer processing capabilities, image processing capabilities, one ormore processors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown. The animationcreation system 730 might be used by animation artists, managers, andothers to specify details, perhaps programmatically and/orinteractively, of imagery to be generated. From user input and data froma database or other data source, indicated as a data store 732, theanimation creation system 730 might generate and output datarepresenting objects (e.g., a horse, a human, a ball, a teapot, a cloud,a light source, a texture, etc.) to an object storage 734, generate andoutput data representing a scene into a scene description storage 736,and/or generate and output data representing animation sequences to ananimation sequence storage 738.

Scene data might indicate locations of objects and other visualelements, values of their parameters, lighting, camera location, cameraview plane, and other details that a rendering engine 750 might use torender CGI imagery. For example, scene data might include the locationsof several articulated characters, background objects, lighting, etc.specified in a two-dimensional space, three-dimensional space, or otherdimensional space (such as a 2.5-dimensional space, three-quarterdimensions, pseudo-3D spaces, etc.) along with locations of a cameraviewpoint and view place from which to render imagery. For example,scene data might indicate that there is to be a red, fuzzy, talking dogin the right half of a video and a stationary tree in the left half ofthe video, all illuminated by a bright point light source that is aboveand behind the camera viewpoint. In some cases, the camera viewpoint isnot explicit, but can be determined from a viewing frustum. In the caseof imagery that is to be rendered to a rectangular view, the frustumwould be a truncated pyramid. Other shapes for a rendered view arepossible and the camera view plane could be different for differentshapes.

The animation creation system 730 might be interactive, allowing a userto read in animation sequences, scene descriptions, object details, etc.and edit those, possibly returning them to storage to update or replaceexisting data. As an example, an operator might read in objects fromobject storage into a baking processor that would transform thoseobjects into simpler forms and return those to the object storage 734 asnew or different objects. For example, an operator might read in anobject that has dozens of specified parameters (movable joints, coloroptions, textures, etc.), select some values for those parameters andthen save a baked object that is a simplified object with now fixedvalues for those parameters.

Rather than have to specify each detail of a scene, data from the datastore 732 might be used to drive object presentation. For example, if anartist is creating an animation of a spaceship passing over the surfaceof the Earth, instead of manually drawing or specifying a coastline, theartist might specify that the animation creation system 730 is to readdata from the data store 732 in a file containing coordinates of Earthcoastlines and generate background elements of a scene using thatcoastline data.

Animation sequence data might be in the form of time series of data forcontrol points of an object that has attributes that are controllable.For example, an object might be a humanoid character with limbs andjoints that are movable in manners similar to typical human movements.An artist can specify an animation sequence at a high level, such as“the left hand moves from location (X1, Y1, Z1) to (X2, Y2, Z2) overtime T1 to T2”, at a lower level (e.g., “move the elbow joint 2.5degrees per frame”) or even at a very high level (e.g., “character Ashould move, consistent with the laws of physics that are given for thisscene, from point P1 to point P2 along a specified path”).

Animation sequences in an animated scene might be specified by whathappens in a live action scene. An animation driver generator 744 mightread in live action metadata, such as data representing movements andpositions of body parts of a live actor during a live action scene, andgenerate corresponding animation parameters to be stored in theanimation sequence storage 738 for use in animating a CGI object. Thiscan be useful where a live action scene of a human actor is capturedwhile wearing mo-cap fiducials (e.g., high-contrast markers outsideactor clothing, high-visibility paint on actor skin, face, etc.) and themovement of those fiducials is determined by the live action processingsystem 722. The animation driver generator 744 might convert thatmovement data into specifications of how joints of an articulated CGIcharacter are to move over time.

A rendering engine 750 can read in animation sequences, scenedescriptions, and object details, as well as rendering engine controlinputs, such as a resolution selection and a set of renderingparameters. Resolution selection might be useful for an operator tocontrol a trade-off between speed of rendering and clarity of detail, asspeed might be more important than clarity for a movie maker to test aparticular interaction or direction, while clarity might be moreimportant that speed for a movie maker to generate data that will beused for final prints of feature films to be distributed. The renderingengine 750 might include computer processing capabilities, imageprocessing capabilities, one or more processors, program code storagefor storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown.

The visual content generation system 700 can also include a mergingsystem 760 that merges live footage with animated content. The livefootage might be obtained and input by reading from the live actionfootage storage 720 to obtain live action footage, by reading from thelive action metadata storage 724 to obtain details such as presumedsegmentation in captured images segmenting objects in a live actionscene from their background (perhaps aided by the fact that the greenscreen 710 was part of the live action scene), and by obtaining CGIimagery from the rendering engine 750.

A merging system 760 might also read data from a rulesets formerging/combining storage 762. A very simple example of a rule in aruleset might be “obtain a full image including a two-dimensional pixelarray from live footage, obtain a full image including a two-dimensionalpixel array from the rendering engine 750, and output an image whereeach pixel is a corresponding pixel from the rendering engine 750 whenthe corresponding pixel in the live footage is a specific color ofgreen, otherwise output a pixel value from the corresponding pixel inthe live footage.”

The merging system 760 might include computer processing capabilities,image processing capabilities, one or more processors, program codestorage for storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. The merging system 760 might operateautonomously, following programming instructions, or might have a userinterface or programmatic interface over which an operator can control amerging process. In some embodiments, an operator can specify parametervalues to use in a merging process and/or might specify specific tweaksto be made to an output of the merging system 760, such as modifyingboundaries of segmented objects, inserting blurs to smooth outimperfections, or adding other effects. Based on its inputs, the mergingsystem 760 can output an image to be stored in a static image storage770 and/or a sequence of images in the form of video to be stored in ananimated/combined video storage 772.

Thus, as described, the visual content generation system 700 can be usedto generate video that combines live action with computer-generatedanimation using various components and tools, some of which aredescribed in more detail herein. While the visual content generationsystem 700 might be useful for such combinations, with suitablesettings, it can be used for outputting entirely live action footage orentirely CGI sequences. The code may also be provided and/or carried bya transitory computer readable medium, e.g., a transmission medium suchas in the form of a signal transmitted over a network.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present.

The use of any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate embodiments ofthe invention and does not pose a limitation on the scope of theinvention unless otherwise claimed. No language in the specificationshould be construed as indicating any non-claimed element as essentialto the practice of the invention.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

Further embodiments can be envisioned to one of ordinary skill in theart after reading this disclosure. In other embodiments, combinations orsub-combinations of the above-disclosed invention can be advantageouslymade. The example arrangements of components are shown for purposes ofillustration and it should be understood that combinations, additions,re-arrangements, and the like are contemplated in alternativeembodiments of the present invention. Thus, while the invention has beendescribed with respect to exemplary embodiments, one skilled in the artwill recognize that numerous modifications are possible.

For example, the processes described herein may be implemented usinghardware components, software components, and/or any combinationthereof. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims and that the invention is intended to cover allmodifications and equivalents within the scope of the following claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

What is claimed is:
 1. A computer-implemented method for generating ananimation control rig configured to manipulate a non-rigid portion of adigital character, the method comprising: associating a control shapewith the non-rigid portion of the digital character; accepting a signalfrom a user input control to move the control shape to specify a newpose for the non-rigid portion of the digital character; traversing ahierarchical node graph representing a plurality of animation controlpoints associated with the new pose; identifying a plurality of nodesthat are implicated in moving the digital character to the new pose; andusing the implicated nodes to display the new pose for the non-rigidportion of the digital character.
 2. The method of claim 1, wherein thedigital character includes a skeleton, wherein the control shape isadapted to handle an area of motion.
 3. The method of claim 1, whereinthe non-rigid portion includes a belly.
 4. The method of claim 1,wherein the non-rigid portion includes a tail.
 5. The method of claim 1,wherein the non-rigid portion includes lips.
 6. The method of claim 5,wherein the non-rigid portion includes a face.
 7. An apparatus includinga processor configured to perform the actions recited in claim
 1. 8. Oneor more non-transitory processor-readable media including instructionsexecutable by one or more processors to perform the actions recited inclaim 1.